Using Adaptive Tracking to Classify and Monitor Activities in a Site

We describe a vision system that monitors activity in a site over extended periods of time. The system uses a distributed set of sensors to cover the site, and an adaptive tracker detects multiple moving objects in the sensors. Our hypothesis is that motion tracking is su cient to support a range of computations about site activities. We demonstrate using the tracked motion data: to calibrate the distributed sensors, to construct rough site models, to classify detected objects, to learn common patterns of activity for di erent object classes, and to detect unusual activities. 1 A motivating scenario Our goal is a vision system that monitors activity in a site over extended periods of time, i.e., patterns of motion and interaction demonstrated by objects in the site. The system should provide statistical descriptions of typical activity patterns, e.g., normal vehicular volume or normal pedestrian tra c paths for a given time of day; it should detect unusual events, by spotting activities that are very di erent from normal patterns, e.g., unusual volumes of tra c, or a speci c movement very di erent from normal observation; and it should detect unusual interactions between objects, e.g., a person parking a car in front of a building, exiting the car, but not entering the building. Because a site may be larger than can be observed by a single camera, our system observes activities with a \forest of sensors" distributed around the site. Each sensor unit is a compact packaging of camera, onboard computational power, local memory, communication capability and possibly locational instrumentation (e.g., GPS). Example systems exist [4, 5, 8], and more powerful systems will emerge as technology in sensor design, DSP processing, and communications evolves. We arbitrarily distribute many sensor units This report describes research supported in part by DARPA under ONR grant N00014-97-0363. around the site, by attaching them to poles, trees, and buildings for outdoor sites, and to walls and furniture for indoor sites, such as the Intelligent Room. The forest should learn patterns of activities in a site, then monitor and classify activities based on these learned patterns. For simplicity, we assume the existence of some basic sensor units, and focus on the processing needed to learn and monitor activities. A coordinated forest of sensors needs: self-calibration { determine the positions of all the cameras relative to one another; construction of rough site models { determine the ground plane, and mark occupied areas; robust detection of objects in the site and classi cation of detected objects; learning from extended observation (e.g. over a period of weeks) the common activity patterns; and detection of unusual events in the site. Our governing hypothesis is that these tasks can be accomplished simply by observing moving objects. To verify this hypothesis, we need: a robust tracker that can reliably detect moving objects and return an accurate description of the observed object, both its motion parameters and its intrinsic parameters such as size and shape; and methods that can use such tracking data to accomplish the tasks listed above. 2 A robust adaptive tracker In this section, we describe a novel tracking system[9], based on the standard notion of background subtraction. Simple implementations of backgrounding just subtract consecutive images and threshold the resulting di erence image to determine pixels that may correspond to motion. More robust methods use time averages of images [2], adaptive Gaussian estimation[12], or Kalman ltering [7] to derive the background image to be subtracted. While such methods often run in real time, they are generally not robust. They often only detect the leading and trailing edges of large objects, they are subject (see http://www.ai.mit.edu/projects/darpa/vsam/) (see http://www.ai.mit.edu/projects/hci/hci.html)